Systems and methods for stereoscopic imaging

ABSTRACT

One or more block configurations are selected, based on object information associated with an object, from a plurality of block configurations having at least one of different sizes or different shapes. A first point in a first image is matched with a corresponding second point in a second image using the one or more block configurations.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of patent application Ser. No.16/240,704, filed on Jan. 4, 2019, which is a continuation ofInternational Application No. PCT/CN2016/088829, filed on Jul. 6, 2016,the entire contents of both of which are incorporated herein byreference.

COPYRIGHT NOTICE

A portion of the disclosure of this patent document contains materialwhich is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by anyone of the patent documentor the patent disclosure, as it appears in the Patent and TrademarkOffice patent file or records, but otherwise reserves all copyrightrights whatsoever.

FIELD

The disclosed embodiments relate generally to computer vision and moreparticularly, but not exclusively, to systems and methods forstereoscopic imaging.

BACKGROUND

High-precision depth mapping is important for applications such asautomatic obstacle avoidance in robotics and mobile platforms,geographic mapping, and space exploration. Stereoscopic imaging is atechnique for providing depth information using binocular vision. Such atechnique can be used for constructing a depth map from two or moreimages by matching features in the images. Existing techniques formatching features, however, often have mismatching problems, especiallywhen encountering complex images with significant depth variation orsmooth texture.

In view of the foregoing, there is a need for systems and methods forstereoscopic imaging that overcome the disadvantages of currentlysystems and methods.

SUMMARY

In accordance with a first aspect disclosed herein, there is set forth amethod of block matching between a first image and a second image forstereoscopic imaging, comprising: selecting a block configuration from aplurality of block configurations; and matching a first point on thefirst image with a corresponding second point on the second image usingthe selected block configuration.

In accordance with another aspect disclosed herein, there is set forth astereoscopic imaging system, comprising: a first imaging deviceconfigured to obtain a first image; a second imaging device configuredto obtain a second image; and one or more processors configured to:select a block configuration from a plurality of block configurations;and match a first point on the first image with a corresponding secondpoint on the second image using the selected block configuration.

In accordance with another aspect disclosed herein, there is set forthan apparatus for stereoscopic imaging, comprising one or more processorsconfigured to: receive a first image; receive a second image; select ablock configuration from a plurality of block configurations; and matcha first point on the first image with a corresponding second point onthe second image using the selected block configuration.

In accordance with another aspect disclosed herein, there is set forth acomputer readable storage medium, comprising: instruction for selectinga block configuration from a plurality of block configurations forstereoscopic imaging; and instruction for matching a first point on thefirst image with a corresponding second point on the second image usingthe selected block configuration.

In accordance with another aspect disclosed herein, there is set forth aprocessing system, comprising: a receiving module for receiving a firstimage obtained by an first imaging device and a second image obtained bya second imaging device; a selecting module for selecting a blockconfiguration from a plurality of block configurations for stereoscopicimaging; and a matching module for matching a first point on the firstimage with a corresponding second point on the second image using theselected block configuration.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an exemplary diagram illustrating an embodiment of a systemfor stereoscopic imaging.

FIG. 2 is an exemplary diagram illustrating an embodiment of blockmatching using the system of FIG. 1.

FIG. 3 is an exemplary diagram illustrating an embodiment of astereoscopic imaging apparatus of the system of FIG. 1.

FIG. 4 is an exemplary top-level flow chart illustrating an embodimentof a method for block matching using the system of FIG. 1.

FIG. 5 is an exemplary diagram illustrating an embodiment of a methodfor triangulation of distance using the system of FIG. 1.

FIG. 6 is an exemplary diagram illustrating another embodiment of amethod for triangulation of distance using the system of FIG. 1.

FIG. 7 is an exemplary diagram illustrating an embodiment of blockmatching using the system of FIG. 1.

FIG. 8 is an exemplary diagram illustrating an embodiment of the systemof FIG. 1, wherein an image pre-processor is shown.

FIG. 9 is an exemplary diagram illustrating an embodiment of a methodfor rectifying images using the image pre-processor of FIG. 8.

FIG. 10 is an exemplary diagram illustrating an embodiment of the systemof FIG. 1, wherein a disparity validator is shown.

FIG. 11 is an exemplary diagram illustrating an embodiment of the systemof FIG. 1, wherein block matching using matching costs are shown.

FIG. 12 is an exemplary diagram illustrating an embodiment of a methodfor identifying corresponding points using the system of FIG. 1.

FIG. 13 is an exemplary flow chart illustrating an embodiment of amethod for identifying corresponding points using the system of FIG. 1.

FIG. 14 is an exemplary diagram illustrating an embodiment of the systemof FIG. 1, wherein matching costs are combined to identify a candidatepoint.

FIG. 15 is an exemplary diagram illustrating an embodiment of the systemof FIG. 1, wherein candidate disparities are found for determining abinocular disparity.

FIG. 16 is an exemplary diagram illustrating another embodiment of thesystem of FIG. 1, wherein candidate disparities are validated using adisparity validator.

FIG. 17 is an exemplary top level block diagram illustrating modules ofa processing system for block matching.

FIG. 18 is exemplary diagram illustrating an embodiment of the system ofFIG. 1, wherein the system is an unmanned aerial vehicle (UAV).

It should be noted that the figures are not drawn to scale and thatelements of similar structures or functions are generally represented bylike reference numerals for illustrative purposes throughout thefigures. It also should be noted that the figures are only intended tofacilitate the description of the embodiments. The figures do notillustrate every aspect of the described embodiments and do not limitthe scope of the present disclosure.

DETAILED DESCRIPTION OF THE EMBODIMENTS

Current systems and methods have drawbacks in ensuring high accuracyblock matching between two images for stereoscopic imaging. The presentsystems and methods improve accuracy in block matching by using blocksof varying shapes and sizes, as described in further specificity below.

The present systems and methods are illustrated with respect to anexemplary stereoscopic imaging system 100 in FIG. 1. The stereoscopicimaging system 100 can include a first imaging device 201 and a secondimaging device 202 for capturing a first image 221 and a second image222, respectively. Although the stereoscopic imaging system 100 is shownto include two imaging devices 201, 202 for illustrative purposes only,the present systems and methods can use any number of imaging devices(for example, one, two, three, four, five, six, or more imaging devices)for stereoscopic imaging, as desired. For example, two or more pairs ofimaging devices can be used for enhanced stereoscopic imaging, asdesired. The imaging devices 201, 202 can receive incident light from anexternal environment and convert the incident light into digital and/oranalog signals (not shown). The signals can be processed to produce theimages 221, 222 that can be displayed to a user. Exemplary imagingdevices 201, 202 suitable for use with the present systems and methodsinclude, but are not limited to, commercially-available cameras andcamcorders.

The imaging devices 201, 202 can be relatively positioned in a selectedmanner, so the images 221, 222 can perceive an object (shown in FIG. 2)from different perspectives. The imaging devices 201, 202 can bepositioned in any desired manner in the stereoscopic imaging system 100.In some embodiments, for example, the imaging devices 201, 202 can bepositioned side-by-side so that the imaging devices 201, 202 haveparallel optical axes.

The stereoscopic imaging system 100 can include a stereoscopic imagingapparatus 300 for receiving the images 221, 222 from the imaging devices201, 202, respectively. The stereoscopic imaging apparatus 300 canobtain depth information of the object based on the images 221, 222.Exemplary depth information can include an object distance from theobject to the first imaging device 201 and/or the second imaging device202.

The stereoscopic imaging apparatus 300 can obtain the depth informationby matching the first image 221 with the second image 222 using blockmatching. As shown in FIG. 1, the stereoscopic imaging apparatus 300 canselect a plurality of block configurations 310 for block matching. Eachblock configuration 310 can include one or more configuration parametersfor defining one or more blocks 341, 342 (shown in FIG. 2) on each ofthe first image 221 and the second image 222, respectively. Exemplaryconfiguration parameters can include size (expressed by number ofpixels, for example), shape, or a combination thereof.

As shown in FIG. 1, a block configuration calculator 320 can beimplemented based on a corresponding block configuration 310. That is,the block configuration calculators 320A, 320B, . . . , 320N can beimplemented based on block configurations 310A, 320B, . . . , 310N,respectively. The block configuration calculator 320 can perform anycalculation and/or operations used for evaluating matching between theblocks 341, 342. For example, evaluating matching can includecalculating a matching cost c. The matching cost c can quantify asimilarity between pixels in the blocks 341, 342. Any suitablecomparison method between pixels of two blocks can be used to find thematching cost c. Exemplary methods for finding the matching cost cinclude comparison of pixel values using correlation, covariance, sum ofabsolute difference, sum of squared differences, and the like.

In some embodiments, results generated by the block configurationcalculators 320 can be inputted to a cross-block-configurationcalculator 330 to compare and contrast matching results using differentblock configurations 310.

Turning now to FIG. 2, exemplary first and second images 221, 222 of anobject 230 are shown for block matching using an exemplary blockconfiguration 310 that is circular. Based on the block configuration310, a first block 341 that includes a first point 241 is defined on thefirst image 221. Similarly, a second block 342 that includes the secondpoint 242 can be defined. Each point 241, 242 can include one or moreadjacent pixels representing a part of the image. In some embodiments,each point can be a single pixel. In some embodiments, each point caninclude a feature (for example, a set of pixels representing an object)represented by a plurality of pixels.

Although FIG. 2 shows the block configuration 310 as having a circularshape for illustrative purposes only, the block configurations 310 canhave any uniform and/or different shapes. At least one of the blockconfigurations 310 can have a non-rectangular shape. Exemplary blockconfigurations 310 can have a triangular shape, a circular shape, anoval shape, a trapezoidal shape, a polygonal shape, or any combinationthereof. The shapes can be regular or irregular, as desired.Additionally and/or alternatively, the block configurations 310 can haveany uniform and/or different sizes. Exemplary dimensions include, forexample, 7 pixels, 9 pixels, 13 pixels, 20 pixels, 25 pixels, or more.In one example, the block configuration 310 can have a square shapehaving a size of 7 pixels by 7 pixels, 9 pixels by 9 pixels, 13 pixelsby 13 pixels, or 25 pixels by 25 pixels.

Turning now to FIG. 3, the exemplary stereoscopic imaging apparatus 300is shown as including processor 302 and a memory 304. The processor 302can be any type of processing system for implementing block matchingfunctions and/or other operations described in the present disclosure.Exemplary processors 302 include, without limitation, one or moregeneral purpose microprocessors (for example, single or multi-coreprocessors), application-specific integrated circuits (ASICs),application-specific instruction-set processors, graphics processingunits (GPUs), physics processing units, digital signal processing units,coprocessors, network processing units, audio processing units,encryption processing units, and the like.

In certain embodiments, the processor 302 can include specializedhardware and/or software for performing various block matching functionsand operations described herein. For example, the processor 302 caninclude an image processing engine or media processing unit, which caninclude specialized hardware for enhancing the speed and efficiency ofcertain operations for image processing and block matching. Specializedhardware can include, but are not limited to, specialized parallelprocessors, caches, high speed buses, and the like. In some embodiments,the processor 302 can include field-programmable gate arrays (FPGAs),ASICs, central processing units (CPUs), GPUs, visual processing units(VPUs), or a combination thereof.

Although FIG. 3 shows the stereoscopic imaging apparatus 300 asincluding one processor 302 for illustrative purposes only, thestereoscopic imaging apparatus 300 can include any number of uniformand/or different processors 302.

The memory 304 (alternatively referred to herein as a non-transitorycomputer readable medium) can comprise any type of memory and can be,for example, a random-access memory (RAM), a static RAM, a dynamic RAM,a read-only memory (ROM), a programmable ROM, an erasable programmableROM, an electrically erasable programmable ROM, a flash memory, a securedigital (SD) card, and the like. In some embodiments, the memory 304 hasa storage capacity that accommodates the needs of the block matchingfunctions and operations described herein. The memory 304 can have anycommercially-available memory capacity suitable for use in imageprocessing applications and, in some embodiments, has a storage capacityof at least 512 Megabytes, 1 Gigabyte, 2 Gigabytes, 4 Gigabytes, 16Gigabytes, 32 Gigabytes, 64 Gigabytes, or more. Although FIG. 3 showsthe stereoscopic imaging apparatus 300 as including one memory 304 forillustrative purposes only, the stereoscopic imaging apparatus 300 caninclude any number of uniform and/or different memories 304.

The stereoscopic imaging apparatus 300 can further include any hardwareand/or software desired for performing the block matching functions andoperations described herein. For example, the stereoscopic imagingapparatus 300 can include one or more input/output interfaces (notshown) for receiving the images 221, 222 shown in FIG. 1. Exemplaryinterfaces include, but are not limited to, universal serial bus (USB),digital visual interface (DVI), display port, serial ATA (SATA), IEEE1394 interface (also known as FireWire), serial, video graphics array(VGA), super video graphics array (SVGA), small computer systeminterface (SCSI), high-definition multimedia interface (HDMI), audioports, and/or proprietary input/output interfaces. As another example,the stereoscopic imaging apparatus 300 can include one or moreinput/output devices (not shown), for example, buttons, a keyboard,keypad, trackball, displays, and/or a monitor. As yet another example,the stereoscopic imaging apparatus 300 can include hardware forcommunication between components of the image processing system 100 (forexample, between the processor 302 and the memory 304).

Turning now to FIG. 4, an exemplary method 400 for stereoscopic imagingis shown. Optionally, the first image 221 can be obtained, at 401.Optionally, the second image 221 can be obtained, at 402. In oneexample, the stereoscopic imaging apparatus 300 can receive the images221, 222 respectively obtained from the imaging devices 201, 202 shownin FIG. 2. In another example, the stereoscopic imaging apparatus 300can obtain the images 221, 222 stored on another computer system (notshown), and the stereoscopic imaging apparatus 300 is not necessarily incommunication with the imaging devices 201, 202.

At 410, a block configuration 310 is selected from a plurality of blockconfigurations 310. For example, the block configuration 310 can beselected from a library of block configurations 310. The library ofblock configurations 310 can be predefined by an operator, as desired.In an illustrative and non-limiting example, the block configuration 310can be selected based on a texture of the object 230 (shown in FIG. 2).In the example, when the object 230 has a smooth texture (such as amonochromatic wall or smooth glass), the block configurations 310 havinga small size may provide limited information and result in mismatching.Thus, a block configurations 310 having a large size can be selected toprovide more information for block matching. In another example, whenthe object 230 has a rough texture with significant depth variation, ablock configuration 310 having a large size can lead to poor overlapping(or large amount of mismatching features) between the images 221, 222.Such poor overlapping can result in a very low degree of matching. Thus,a block configuration 310 having a small size can be selected. In stillanother example, a block configuration 310 having a shape and/or sizematching shape of the object 230 or a portion of the object 230 can beselected, for the block configuration 310 to include a selected texturefor matching.

In one embodiment, the stereoscopic imaging apparatus 300 can select oneor more predetermined block configurations 310. For example, an operatorcan select block configurations 310 with a variety of shapes and/orsizes, with or without consideration of content of the images 221, 222.

In some embodiments, the stereoscopic imaging apparatus 300 can selectthe block configuration 310 dynamically. For example, blockconfigurations 310 can be selected based on content of the images 221,222, such as based on the shape and/or size of object 230 in the images221, 222. For example, the stereoscopic imaging apparatus 300 canperform one or more image feature recognition and/or feature detectionmethods to determine texture of the object 230 in the images 221, 222.Exemplary feature recognition methods can include edge detection (forexample, Shi & Tomasi corner detection), image binarization, SURF blob(Speeded Up Robust Features) detection, MSER blob (Maximally StableExtremal Regions) detection, SIFT (Scale-Invariant Feature Transform)descriptors, FREAK (Fast REtinA Keypoint) descriptors, BRISK (BinaryRobust Invariant Scalable Keypoints) descriptors, HOG (Histogram ofOriented Gradients) descriptors, and the like.

In yet other embodiments, the stereoscopic imaging apparatus 300 canselect the block configuration 310 randomly. Randomizing the selectionof the block configuration 310 can reduce matching error associated withusing a fixed set of block configuration 310 for a specific type oftexture, for example.

As shown in FIG. 4, at 420, the first point 241 on the first image 221is matched with the corresponding second point 242 on the second image222 using the block configurations 310. Stated somewhat differently, thesecond point 242 that corresponds to the first point 241 can beidentified based on the matching.

The method 400 can advantageously improve accuracy of block matching. Aspreviously described, different size and/or shape of the blockconfiguration 310 can result in different accuracy of block matching.When a single block configuration 310 is used, the block configuration310 may not be able to adapt to all types of complex scenes or objects.Using a plurality of block configurations 310 can ensure that one ormore block configurations 310 can be suitable for achieving accuratematching and reducing probability of mismatching. Using the plurality ofblock configurations 310 can thus compensate for limitations orinaccuracies associated with single block configuration 310, therebyimproving the accuracy and reliability of a block matching result.

FIG. 5 and FIG. 6 are shown to illustrate obtaining the depthinformation using the images 221, 222. FIG. 5 shows the imaging devices201, 202 imaging the object 230 using stereopsis. FIG. 6 illustrates amethod of ascertaining an object distance D using triangulation withreference to the two imaging devices 201, 202. Each of the imagingdevices 201, 202 perceives a same point 220 on the object 230, but indifferent spatial coordinates as illustrated by the coordinate axes (x1,y1, z1) and (x2, y2, z2). The imaging devices 201, 202 perceive thepoint 220 along respective optical axes 261, 262 and thereby arrive attwo different images 221, 222 of the point 220. The point 220 iscaptured in the images 221, 222 as the first and second points 241, 242(shown in FIG. 2), respectively. The images 221, 222 can betwo-dimensional. The images 221, 222 are typically different, beingtaken from different positions, unless the imaging devices 201, 202 arepositioned such that their optical axes 261, 262 coincide. Accordingly,under most circumstances, a disparity d shown in Equation (4) can befound between the images 221, 222, as described below with reference toFIG. 6.

Referring now to FIG. 6, the images 221, 222 can be compared toascertain the object distance D between the pair of imaging devices 201,202 and the point 220. A method of triangulation can be used toascertain the object distance D using the disparity d shown in Equation(4). Specifically, the position of the point 220 having an index i,represented by coordinates (x_(i), y_(i), z_(i)), can be given asfollows:

$\begin{matrix}{{x_{i} = {\frac{b}{d}\left( {x_{i}^{K} - c_{x}} \right)}},} & {{Equation}\mspace{14mu}(1)} \\{{y_{i} = {\frac{b}{d}\left( {y_{i}^{K} - c_{y}} \right)}},{and}} & {{Equation}\mspace{14mu}(2)} \\{z_{i} = {\frac{b}{d}f}} & {{Equation}\mspace{14mu}(3)}\end{matrix}$

where cx and cy represent respective center coordinates of the firstimaging device 201 (or the second imaging device 202), x_(i) ^(K) andy_(i) ^(K) represent the coordinates representing the point 220 in thefirst image 221 where K=1 (or the second image 222 where K=2), b is thebaseline (in other words, the distance between the center coordinates ofthe imaging devices 201, 202), ƒ is the focal length of each of theimaging devices 201, 202 (assuming here that the imaging devices 201,202 have the same focal length), i is an index over multiple points 220of the object 230, and d is the disparity between the images 221, 222,represented here as:

d _(i) =x _(i) ¹ −x _(i) ²  Equation (4)

Based on the images 221, 222 and using Equations (1)-(4), thestereoscopic imaging apparatus 300 shown in FIG. 1 can determine thecoordinate z_(i). The object distance D can be equal to and/or based onthe coordinate z_(i).

Turning to now FIG. 7, another illustrative example of matching pointsbetween two images is shown. As shown in FIG. 7, for a given first 241on the first image 221, a matching process can be used to select asecond point 242 on the second image 222 that corresponds to the firstpoint 241. A plurality of candidate second points 282 a . . . 282 m canbe selected for evaluation of a matching cost c for determining thesuitability of matching. As described above with reference to FIG. 1,any suitable comparison method between pixels of two blocks can be usedto find the matching cost c. Exemplary methods for finding the matchingcost c include comparison of pixel values using correlation, covariance,sum of absolute difference, sum of squared differences, and the like.Based on the matching costs c, the second point 242 can be selected fromamong the candidate second points 282 a . . . 282 m.

The candidate second points 282 a . . . 282 m for matching can beselected in any manner. In some embodiments, the candidate second points282 a . . . 282 m can be selected in a predefined neighborhood about thecoordinates of the first point 241. For example, candidate second points282 a . . . 282 m can be evaluated that are within a defined range thatis 2 pixels, 4 pixels, 8 pixels, 16 pixels, 32 pixels, 64 pixels, 128pixels, 256 pixels, or more of the coordinates of the first point 241.Alternatively, all pixels of the second image 222 can be evaluated. Insome embodiments, all of the pixels within the defined range can beevaluated for the matching cost c. In other embodiments, pixels withinthe defined range can be sampled (for example, randomly or by intervalsampling) for the matching cost c.

Turning now to FIG. 8, an exemplary stereoscopic imaging system 100 caninclude an image pre-processor 350 for pre-processing images 221, 222.The image pre-processor 350 can execute any image transformationoperations for projecting the images 221, 222 onto a common image planedefined in a common coordinate system (shown in FIG. 9). For example,the image pre-processor 350 can be used for rectifying the images 221,222. The image pre-processor 350 can transform the images 221, 222 sothat the coordinate axes (x1, y1, z1) and (x2, y2, z2) are aligned. Forexample, the image pre-processor 350 can perform rotation to align thex1-axis with the x2-axis and/or align the y1-axis with the y2-axis.Additionally and/or alternatively, the image pre-processor 350 canperform scaling transformation to eliminate scaling difference betweenthe images 221, 222. Additionally and/or alternatively, the imagepre-processor 350 can perform rotation and/or skew adjustments in zdirection to align the z1-axis with the z2-axis. By pre-processing theimages 221, 222 images, the pre-processor 350 can simplify subsequentblock matching between the images 221, 222, advantageously improvingmatching efficiency.

Additionally and/or alternatively, the image pre-processor 350 can beconfigured to perform any image processing operations to improve qualityof the images 221, 222. Subsequent block matching can advantageously bemore easily performed. Exemplary image processing operations include,but are not limited to, image de-noising, image sharpening, and/or imagede-blurring.

Although FIG. 8 shows the image pre-processor 350 and the stereoscopicimaging apparatus 300 as being individual units, the image pre-processor350 and the stereoscopic imaging apparatus 300 can be at least partiallyintegrated, without limitation.

Turning now to FIG. 9, the images 221, 222 after rectification areshown. Rectification advantageously aligns the images 221, 222 along thesame y-axis and z-axis, such that the images 221, 22 are offset onlyalong the x-axis. Stated somewhat differently, the second point 242 canhave a same y-axis coordinate as the first point 241. Thus, thecandidate second points 282 can be chosen in the second image 222 alonga horizontal line 226 that passes the first point 241. Searching of thecandidate second points 282 can thus be one-dimensional. Computation forthe searching is advantageously simplified.

After rectification of the images 221, 222, and/or other pre-processingsteps, the image pre-processor 350 (shown in FIG. 8) can transmit therectified images 221, 222 to the stereoscopic imaging apparatus 300.Subsequently, a block configuration 310 can be selected from a pluralityof block configurations. Points 241, 242 on the rectified images 221,222 can be matched using the selected block configuration 310 using thepresent systems and methods.

Turning to FIG. 10, the stereoscopic imaging apparatus 300 can include adisparity validator 360. The disparity validator 360 can perform anycalculation operations used for evaluating accuracy of the disparity dusing a predetermined criteria. When accuracy of the disparity d doesnot satisfy the predetermined criteria, the disparity validator 360 candetermine that the disparity d is invalid. Although FIG. 10 shows thedisparity validator 360 and the cross-block-configuration calculator 330as being individual units, the disparity validator 360 and thecross-block-configuration calculator 330 can be at least partiallyintegrated, without limitation.

Turning to FIG. 11, exemplary embodiments are now shown to illustrategenerating the matching cost c to match the points 241, 242 (shown inFIG. 2) for obtaining the disparity d. Each block configurationcalculator 320 can generate an individual matching cost MC_(ij) betweenthe first point 241 and a selected candidate second point 282 shown inFIG. 7. In the individual matching cost MC_(ij), i is an index over theblock configurations 310 and j is an index over the candidate secondpoints 282. Stated somewhat differently, between the first point 241 andthe jth candidate second point 282, block configuration calculators320A, 320B, . . . , 320N can calculate individual matching cost MC_(Aj),MC_(Bj), . . . , MC_(Nj), respectively. The individual matching costMC_(ij) can be provided to the cross-block-configuration calculator 330to be combined for identifying the second point 242 corresponding to thefirst point 241.

Turning now to FIG. 12, the first block 341 and the second block 342 areshown to include the first point 241 and the candidate second point 282a. The first block 341 and the second block 342 can be based on theblock configuration 310A. The block configuration 310A can have atriangular shape shown in FIG. 11 for illustrative purposes only. Theindividual matching cost MC_(Aj) between the first point 241 and thecandidate second point 282 a can thus be calculated using the blocks341, 342 based on the block configuration 310A.

Turning now to FIG. 13, an exemplary method 500 for determining anindividual matching cost MC_(ij) for each block configuration 310 isshown. The first block 341 of pixels around the first point 241 of thefirst image 221 can be chosen, at 510, based on the block configuration310. The block configuration 310 can include any one of the blockconfigurations 310A, 310B, . . . , 310N shown in FIG. 11. A second block342 of pixels around the candidate second point 282 of the second image222 can be chosen based on the block configuration 310. The candidatesecond point 282 can include any one of the candidate second points 282a, 282 b, . . . , 282 m shown in FIG. 12.

The individual matching cost MC_(ij) for the block configuration 310between the first point 241 of the first image 221 and the selectedcandidate second point 282 of the second image 222 can be determined, at530. The method 500 can be performed for each candidate second point 282based on each block configuration 310.

Turning to FIG. 14, the individual cost MC_(ij) is shown to be combinedvia the cross-block-configuration calculator 330 (shown in FIG. 11) toidentify the second point 242 among the candidate second points 282. Asshown in FIG. 14, for the candidate second point 282 a, the individualmatching costs MC_(ij) can include MC_(Aa), MC_(Ba), . . . , MC_(Na).For the candidate second point 282 m, the individual matching costsMC_(ij) can include MC_(Am), MC_(Bm), . . . , MC_(Nm).

For each candidate second point 282, the corresponding individualmatching cost MC_(ij) can be combined to generate the matching cost MG.As shown in FIG. 14, for the candidate second point 282 a, theindividual matching costs MC_(Aa), MC_(Ba), . . . , MC_(Na) can becombined to generate matching cost MC_(a). For the candidate secondpoint 282 m, the sub-individual matching costs MC_(Am), MC_(Bm), . . . ,MC_(Nm) can be combined to generate matching cost MC_(m).

The matching cost MC_(j) of the corresponding second point 282 can begenerated in any suitable manner. For example, the matching cost MC_(j)can be a function ƒ of the individual matching costs MC_(ij) of thecorresponding candidate second point 282:

MC_(j)=ƒ(MC_(Aj),MC_(Bj), . . . ,MC_(Nj)).  Equation (5)

In one embodiment, the matching cost MC_(j) can be determined byselecting one of the individual matching costs MC_(Aj), MC_(Bj), . . . ,MC_(Nj). For example, the matching cost MC_(j) can be determined byselecting the greatest or best one of the individual matching costsMC_(Aj), MC_(Bj), . . . , MC_(Nj), that is, by selecting the individualmatching cost that indicates the best matching.

In another embodiment, the MC_(j) can be determined by weighing theindividual matching costs MC_(Aj), MC_(Bj), . . . , MC_(Nj), asexpressed as follows for illustrative purposes:

MC_(j)=Σ_(i=A) ^(N) w _(i)MC_(ij)  Equation (6)

where w_(i) is a weight of the ith block configuration 310. The w_(i)can be pre-determined based on accuracy of the ith block configuration310 for all images or for the specific type of content of images 221,222 shown in FIG. 12.

The matching costs MC_(j) of the candidate second points 282 can becompared to identify a candidate second point 282. In one example, thesecond point 242 having the greatest matching cost MC_(j) can beselected as the matching second point 242. In another example, thematching cost MC_(j) of each candidate second point 282 can be comparedwith a threshold matching cost. When the matching cost MC_(j) indicatesa better match than the threshold matching cost, the correspondingcandidate second point 282 can be selected as the matching second point242.

As shown in FIG. 14, by using the plurality of block configurations310A, 310B, . . . , 310N, a plurality of individual matching costsMC_(ij) can be obtained for each of the candidate second point 282.Problems of mismatch and/or poor match for a single block configuration310 can thereby be prevented. The identified second point 242 can havethe best matching cost that is provided by the block configuration 310that is optimally adapted to the scene of the images 221, 222.

Turning to FIG. 15, each block configuration calculator 320 can generatea candidate disparity d_(CA). Each candidate disparity d_(CA) can beobtained using the second point 242 (shown in FIG. 2) that is identifiedbased on the block configuration 310. Stated somewhat differently, basedon a selected block configuration 310, a second point 242 can beidentified among the candidate second points 282. The candidatedisparity d_(CA) can thus be obtained based on the second point 242. Forexample, the candidate disparities d_(A), d_(B), . . . , d_(N) shown inFIG. 15 can be obtained based on the second points 242 respectivelyidentified using the block configuration 310A, 310B, . . . , 310N.

Using a selected block configuration 310 with index of i, the blockconfiguration calculator 320 can identify the corresponding second point242 in any suitable manner. For example, individual matching costMC_(ij) can be calculated for each candidate second point 282 based onthe ith block configuration 310. The candidate second point 282associated with the best individual matching cost MC_(ij) can beselected as the second point 242 for the ith block configuration 310.

The candidate disparities d_(CA) are inputted to thecross-block-configuration calculator 330 to be combined for identifyingthe disparity d. Turning to FIG. 16, an illustrative example ofcombining the candidate disparities d_(CA) is shown. The candidatedisparities d_(CA) can be validated in any manner. For example, thecandidate disparities d_(CA) can be validated via the disparityvalidator 360. Validation of the candidate disparities d_(CA) of aselected block configuration 310 shown in FIG. 15 can include, forexample, comparing the threshold matching cost with the individualmatching cost MC_(ij) of the second point 242 identified using theselected block configuration 310. If the individual matching costMC_(ij) of the identified second point 242 is worse than the thresholdmatching cost, the candidate disparities d_(CA) associated with such asecond point 242 is not valid.

In an illustrative and non-limiting example, a candidate disparityd_(CA) associated with the block configuration 310 having the greatestsize can be validated first. When the candidate disparity d_(CA) isvalid, the d_(CA) can be identified as the disparity d. A blockconfiguration 310 having a large size can provide great amount ofinformation for matching and thus allow block matching at high accuracy.Thus, starting validation from the block configuration 310 having thegreatest size can advantageously identify the disparity d accurately atearly stage of the validation.

When such a candidate disparity d_(CA) is invalid, a candidate disparityd_(CA) associated with a different block configuration 310 can bevalidated. For example, the candidate disparity d_(CA) of the blockconfiguration 310 having the greatest size can be invalid due to poormatching resulting from significant depth variation of imaging. In thatcase, the block configuration 310 having the same size but a differentshape can be validated. Additionally and/or alternatively, the blockconfiguration 310 having a smaller size with a different and/or sameshape can be validated. Advantageously, by using the plurality of blockconfigurations 310, a probability of successful block matching can beimproved and failed block matching can be prevented.

The validation can be implemented until at least one candidate disparityd_(CA) is identified. Additionally and/or alternatively, the validationcan be implemented on all candidate disparities d_(CA). In some cases,one or more candidate disparities d_(CA) can be determined as valid, andeach of the valid candidate disparities d_(CA) can be identified as thedisparity d. The valid candidate disparities d_(CA) can be combined toform the disparity d, for example, via a filtering technique such asmean filtering and/or median filtering.

Turning now to FIG. 17, an exemplary processing system 600 is shown asincluding one or more modules to perform any of the disclosed methods.The processing system 600 is shown as including a receiving module 610,a selecting module 620, and a matching module 630. In some embodiments,the receiving module 610 can be configured to obtain a first image 221and a second image 222 (both shown in FIG. 1). The receiving module 610can receive the images 221, 222 respectively obtained by a first imagingdevice 201 and a second imaging device 202 (both shown in FIG. 1). Theselecting module 620 can be configured to select a plurality of blockconfigurations 310. The matching module 630 can be configured to matchthe first point 241 on the first image 221 with the corresponding secondpoint 242 on the second image 222 using the block configurations 310, asshown in FIG. 2.

In some embodiments, the block configurations 310 can have differentsizes. In some embodiments, at least one block configuration 310 canhave a size of 7 pixels by 7 pixels, 9 pixels by 9 pixels, 13 pixels by13 pixels, or 25 pixels by 25 pixels. In some embodiments, the blockconfigurations 310 can have different shapes. In some embodiments, atleast one of the block configurations 310 can have a non-rectangularshape. In some embodiments, the block configurations 310 can include ablock configuration 310 with a circular shape, a block configurationwith a triangular shape, a block configuration with a trapezoidal shape,or a combination thereof.

In some embodiments, the selecting module 620 can be configured toselect one or more pre-determined block configurations 310. In someembodiments, the selecting module 620 can be configured to dynamicallyselect the block configurations 310. In some embodiments, the selectingmodule 620 can be configured to randomly select the block configurations310.

In some embodiments, the matching module 630 can be configured todetermine a matching cost c between the first point 241 on the firstimage 221 and each of a plurality of candidate second points 282 (shownin FIG. 7) of the second image 222 to identify the corresponding secondpoint 242. In some embodiments, the matching module 630 can beconfigured to determine a matching cost between the first point 241 onthe first image 221 and each of a plurality of candidate second points282 on a line of the second image 222 to identify the correspondingsecond point 242. In some embodiments, the matching module 630 can beconfigured to determine the matching cost based on individual matchingcosts c for each of the block configurations 310. In some embodiments,the matching module 630 can be configured to determine the matching costby selecting one of the individual matching costs c. In someembodiments, the matching module 630 can be configured to determine thematching cost by weighing the individual matching costs c. In someembodiments, the matching module 630 can be configured to determine thematching cost using a predetermined weight for each block configuration310. In some embodiments, the matching module 630 can be configured todetermine the individual matching costs for each block configuration 310of the block configurations 310 by choosing a first block 341 of pixelsaround the first point 241 of the first image 221 based on the blockconfiguration 310, choosing a second block 342 of pixels around aselected candidate second point 282 of the second image 222 based on theblock configuration 310, and determining the individual matching costfor the block configuration 310 between the first point 241 of the firstimage 221 and the selected candidate second point 282 of the secondimage 222.

In some embodiments, the matching module 630 can be configured todetermine an object distance D based on a disparity d between the firstpoint 241 of the first image 221 and the corresponding second point 242of the second image 222.

In some embodiments, the matching module 630 can be configured todetermine individual matching costs between the first point on the firstimage and each of a plurality of candidate second points on the secondimage based on each of the block configurations to select candidatesecond points respectively corresponding to the block configurations.

In some embodiments, the matching module 630 can be configured to obtaincandidate disparities d_(CA) between the first point 241 and theselected candidate second points 282, respectively, and determine thedisparity d based on the candidate disparities d_(CA). In someembodiments, the matching module 630 is configured to determine anobject distance D based on the disparity d.

In some embodiments, the matching module 630 can be configured to matcha first pixel on the first image 221 with a corresponding second pixelon the second image 222 using the block configurations 310. In someembodiments, the matching module 630 is configured to match a firstfeature on the first image 221 with a corresponding second feature onthe second image 222 using the block configurations 310.

As shown in FIG. 17, the processing system 600 can optionally include apre-processing module 640 for pre-processing the first and second images221, 222 prior to the matching. In some embodiments, the pre-processingmodule 640 is configured to rectify the second image 222 relative to thefirst image 221.

Turning now to FIG. 18, an exemplary stereoscopic imaging system 100 isshown as being a mobile platform that is an unmanned aerial vehicle(UAV) 150. Stated somewhat differently, the stereoscopic imaging system100 can be mounted on the mobile platform, such as the UAV 150. UAVs150, colloquially referred to as “drones,” are aircraft without a humanpilot onboard the vehicle whose flight is controlled autonomously or bya remote pilot (or sometimes both). UAVs are now finding increased usagein civilian applications that require various forms of aerialdata-gathering. Various types of UAVs 150 are suitable for use asstereoscopic imaging systems 100. One suitable type of UAV 150, forexample, is an aerial rotorcraft that is propelled by multiple rotors.One suitable type of rotorcraft has four rotors and is known as aquadcopter, quadrotor helicopter, or quad rotor. Exemplary quadcopterssuitable for the present systems and methods for imaging (for example,stereoscopic imaging) include numerous models currently availablecommercially. UAVs 150 suitable for the present systems and methodsfurther include, but are not limited to, other rotor designs such assingle rotor, dual rotor, trirotor, hexarotor, and octorotor designs.Fixed wing UAVs 150 and hybrid rotorcraft-fixed wing UAVs 150 can alsobe used. Imaging devices 201, 202 can be mounted to the UAV 150 in anysuitable manner (for example, on an underside of the UAV 150). Othercomponents of the stereoscopic imaging system 100 can be included in afuselage of the UAV 150, as appropriate, for protection against wear andtear.

The disclosed embodiments are susceptible to various modifications andalternative forms, and specific examples thereof have been shown by wayof example in the drawings and are herein described in detail. It shouldbe understood, however, that the disclosed embodiments are not to belimited to the particular forms or methods disclosed, but to thecontrary, the disclosed embodiments are to cover all modifications,equivalents, and alternatives.

What is claimed is:
 1. A block matching method comprising: selecting,based on object information associated with an object, one or more blockconfigurations from a plurality of block configurations having at leastone of different sizes or different shapes; and matching, using the oneor more block configurations, a first point in a first image with acorresponding second point in a second image.
 2. The method of claim 1,wherein: the object information is indicative of a texture of theobject; and sizes in the one or more block configurations are negativelycorrelated to a roughness of the texture indicated by the objectinformation.
 3. The method of claim 1, wherein selecting the one or moreblock configurations comprises selecting a block configuration having ashape that matches at least one of a shape or a size of the object or aportion of the object, such that the selected block configurationmatches a texture of the object or a texture of the portion of theobject.
 4. The method of claim 1, wherein: selecting the one or moreblock configurations comprises selecting a first block configuration anda second block configuration from the plurality of block configurationsthat have at least one of different shapes or different sizes; andmatching the first point with the corresponding second point comprises:obtaining a first matching cost by matching, using the first blockconfiguration, the first point in the first image with the correspondingsecond point in the second image; obtaining a second matching cost bymatching, using the second block configuration, the first point in thefirst image with the corresponding second point in the second image; andmatching the first point in the first image with the correspondingsecond point in the second image based on the first matching cost andthe second matching cost.
 5. The method of claim 1, wherein selectingthe one or more block configurations comprises dynamically selecting theone or more block configurations from the plurality of blockconfigurations.
 6. The method of claim 1, wherein matching the firstpoint with the corresponding second point comprises: determiningmatching costs between the first point in the first image and aplurality of candidate second points in the second image; andidentifying the corresponding second point from the plurality ofcandidate second points based on the matching costs.
 7. The method ofclaim 6, wherein determining the matching costs between the first pointand the plurality of candidate second points comprises at least one of:determining matching costs between the first point in the first imageand a plurality of candidate second points on a line of the second imageto identify the corresponding second point; or determining, using oneblock configuration from the block configurations, individual matchingcosts between the first point and the plurality of candidate secondpoints to determine at least one matching cost that includes at leastone of: selecting the at least one matching cost from the individualmatching costs; determining the at least one matching cost based onweighted individual matching costs by applying first weights to theindividual matching costs; or determining the at least one matching costbased on weighted block configurations by applying second weights to theplurality of block configurations.
 8. The method of claim 1, furthercomprising: determining an object distance based on a disparity betweenthe first point in the first image and the corresponding second point inthe second image.
 9. The method of claim 1, wherein matching the firstpoint with the corresponding second point comprises determining, usingeach block configuration of the plurality of block configurations,individual matching costs between the first point and each of aplurality of candidate second points in the second image to selectcandidate second points respectively corresponding to the plurality ofblock configurations.
 10. The method of claim 9, further comprising:obtaining candidate disparities between the first point and the selectedcandidate second points, respectively; determining a disparity betweenthe first point and the corresponding second point based on thecandidate disparities; and determining an object distance based on thedisparity.
 11. The method of claim 1, wherein matching the first pointwith the corresponding second point comprises at least one of: matching,using the one or more block configurations, a first pixel in the firstimage with a corresponding second pixel in the second image; ormatching, using the one or more block configurations, a first feature inthe first image with a corresponding second feature in the second image.12. A stereoscopic imaging system comprising: a first imaging deviceconfigured to obtain a first image; a second imaging device configuredto obtain a second image; and one or more processors configured to:select, based on object information associated with an object, one ormore block configurations from a plurality of block configurationshaving at least one of different sizes or different shapes; and match,using the one or more block configurations, a first point in the firstimage with a corresponding second point in the second image.
 13. Thestereoscopic imaging system of claim 12, wherein: the object informationis indicative of a texture of the object; and sizes in the one or moreblock configurations are negatively correlated to a roughness of thetexture indicated by the object information.
 14. The stereoscopicimaging system of claim 12, wherein the one or more processors arefurther configured to select a block configuration having a shape thatmatches at least one of a shape or a size of the object or a portion ofthe object, such that the selected block configuration matches a textureof the object or a texture of the portion of the object.
 15. Thestereoscopic imaging system of claim 12, where the one or moreprocessors are further configured to: select a first block configurationand a second block configuration from the plurality of blockconfigurations that have a least one of different shapes or differentsizes; obtain a first matching cost by matching, using the first blockconfiguration, the first point in the first image with the correspondingsecond point in the second image; obtain a second matching cost bymatching, using the second block configuration, the first point in thefirst image with the corresponding second point in the second image; andmatch the first point in the first image with the corresponding secondpoint in the second image based on the first matching cost and thesecond matching cost.
 16. The stereoscopic imaging system of claim 12,wherein the one or more processors are further configured to: determinematching costs between the first point in the first image and aplurality of candidate second points in the second image; and identifythe corresponding second point from the plurality of candidate secondpoints based on the matching costs.
 17. The stereoscopic imaging systemof claim 12, the one or more processors are further configured todetermine an object distance based on a disparity between the firstpoint in the first image and the corresponding second point in thesecond image.
 18. The stereoscopic imaging system of claim 12, whereinthe one or more processors are further configured to determine, usingeach block configuration of the plurality of block configurations,individual matching costs between the first point and each of aplurality of candidate second points in the second image to selectcandidate second points respectively corresponding to the plurality ofblock configurations.
 19. The stereoscopic imaging system of claim 18,wherein the one or more processors are further configured to: obtaincandidate disparities between the first point and the selected candidatesecond points, respectively; determine a disparity between the firstpoint and the corresponding second point based on the candidatedisparities; and determine an object distance based on the disparity.20. The stereoscopic imaging system of claim 12, wherein the one or moreprocessors are further configured to: match, using the one or more blockconfigurations, a first pixel in the first image with a correspondingsecond pixel in the second image; and match, using the one or more blockconfigurations, a first feature in the first image with a correspondingsecond feature in the second image.